add TSV-Merge, Iso-C, Subspace Boosting as merge methods. #639
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
|
TSV-Merge: https://arxiv.org/abs/2412.00081 |
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| import time | ||
| import logging | ||
|
|
||
| def iso_c(task_vectors: List[torch.Tensor], tv_key: str, device: torch.device) -> Dict[str, Any]: |
There was a problem hiding this comment.
The function signature indicates a return type of Dict[str, Any], but the implementation returns a torch.Tensor. The return type annotation should be updated to match the actual implementation:
def iso_c(task_vectors: List[torch.Tensor], tv_key: str, device: torch.device) -> torch.Tensor:This will ensure type consistency and help with static type checking.
| def iso_c(task_vectors: List[torch.Tensor], tv_key: str, device: torch.device) -> Dict[str, Any]: | |
| def iso_c(task_vectors: List[torch.Tensor], tv_key: str, device: torch.device) -> torch.Tensor: |
Spotted by Graphite Agent
Is this helpful? React 👍 or 👎 to let us know.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
| S_damped = torch.clamp(S, min=S[cutoff_idx]) | ||
| else: # Clamping approach using the threshold as an index | ||
| cutoff_idx = int(thresh * S.numel()) | ||
| S_damped = torch.clamp(S, min=S[cutoff_idx]) |
There was a problem hiding this comment.
Bug: Undefined variable causing NameError in subspace_boosting
The subspace_boosting function's else branch (when cumsum is False) uses an undefined thresh variable, causing a NameError; svd_thresh was likely intended. Additionally, in the if cumsum: block's fallback, if no SVD cutoff is found, S is clamped to its smallest singular value. This doesn't modify S, contrary to the full spectrum intent.
| ) | ||
| ).to(original_dtype) # Convert back to original dtype | ||
|
|
||
| return new_vector |
There was a problem hiding this comment.
Bug: Inconsistent Init Triggers SVD Runtime Errors
In compute_and_sum_svd_mem_reduction, new_vector is inconsistently initialized as a dict then used for tensor operations, causing runtime errors. The SVD accumulation tensors (sum_u, sum_s, sum_v) are incorrectly sized and can receive zero-length slices from reduced_index_s, leading to out-of-bounds errors. Also, the final SVD condition check uses the last task_vector.
1 participant
Note
Introduce TSV-M, ISO-C, and Subspace Boosting merge methods, wiring SVD-based operations and new parameters into GTA flow.
tsvm,iso_c,task_arithmetic_sb, andties_sbtoSTATIC_MERGE_METHODSwith names, pretty names, and reference URLs.iso_c,compute_and_sum_svd_mem_reduction,subspace_boosting).svd_threshandcumsumto method config and task wiring.GTATask.execute, apply method-specific mixing:iso_c:iso_c(...)tsvm:compute_and_sum_svd_mem_reduction(...)task_arithmetic_sb/ties_sb:subspace_boosting(..., svd_thresh, cumsum)mergekit/subspace_helpers.py)iso_c(mean SVD spectrum equalization),compute_and_sum_svd_mem_reduction(concatenated low-rank SVD merge), andsubspace_boosting(SVD-based damping for selected weights).Written by Cursor Bugbot for commit 3f031f3. This will update automatically on new commits. Configure here.